Adobe-MIT submission to the DSTC 4 Spoken Language Understanding pilot task

The Dialog State Tracking Challenge 4 (DSTC 4) proposes several pilot tasks. In this paper, we focus on the spoken language understanding pilot task, which consists of tagging a given utterance with speech acts and semantic slots. We compare different classifiers: the best system obtains 0.52 and 0.67 F1-scores on the test set for speech act recognition for the tourist and the guide respectively, and 0.52 F1-score for semantic tagging for both the guide and the tourist. 1 Speech act recognition Recognizing the speech acts of the current utterance is one of the two goals of the spoken language understanding pilot task. In the training and development sets, each utterance is annotated with one speech act. One speech act is composed of zero, one or two speech act categories. Each speech act category has in turn zero, one or two speech act attributes. There are 4 speech act categories, and 22 speech act attributes. [6] and [7] give further details on the task. The main approaches for this task are presented in [15, 1, 17, 5, 16, 19, 10, 3]. We submitted 5 systems. Systems 3 and 5 were the best performing ones. System 3 is based on a support vector machine (SVM) classifier to recognize the speech acts: the features are the 5000 most common unigrams, bigrams, trigrams, as well as a binary feature indicating whether the current speaker is different from the speaker in the last utterance. To account for the history, each feature is computed for both the current and the previous utterance. Two SVM classifiers were trained: one for each speaker. The kernel function as well as the penalty parameter of the error term were both optimized with 5-fold cross-validation. System 5 is similar, but with logistic regression as the classifier; moreover, it uses one single speaker-independent model instead of one model per speaker, as it slightly improves the results on the development set. Systems 3 and 5 assume that each utterance contains exactly one speech act category and one speech act attribute: they are therefore multiclass, monolabel classifiers, with 88 possible classes (4 speech act categories×22 speech act attributes). Franck Dernoncourt Adobe Research, San Jose, CA, USA and MIT, Cambridge, MA, USA e-mail: [email protected] Ji Young Lee Massachusetts Institute of Technology, Cambridge, MA, USA e-mail: [email protected] Trung H. Bui Adobe Research, San Jose, CA, USA e-mail: [email protected] Hung H. Bui Adobe Research, San Jose, CA, USA e-mail: [email protected]